1. Field of the Invention
The present invention is a method and system to provide a face-based automatic ethnicity recognition system that utilizes probabilistic graphical models on ethnicity-dependent image features.
2. Background of the Invention
Automatic recognition of demographics using images has a wide range of applications, such as security, marketing, and computer user interface. Real-time applications, such as computer user interface or demographics-targeted advertisements, especially demand highly accurate demographics recognition capabilities.
Ethnicity recognition can enable such applications, for products or services to better serve the needs of individuals based on their ethnic background. Gender recognition is also representative of demographics recognition application, and shares some common characteristics with ethnicity recognition. Both the gender and ethnicity of a person can be derived from certain features of the facial image. Image analysis technologies that work for gender recognition, such as statistical pattern recognition methods, can be utilized for ethnicity recognition as well. However, the task of ethnicity recognition involves unique challenges that the task of gender recognition does not possess.
First, the notion of ethnicity comes from biological, cultural, or historical context, and is not as well-defined as the notion of gender. There are certain genetic elements that define ethnicity, but the notion of an ethnic group is only loosely defined, and often there is no clear boundary between ethnic groups. Inexperienced human observers often make mistakes determining ethnicity solely by appearance. Any automatic means for determining ethnicity will also suffer from such ambiguities. However, it is still feasible to design an ethnicity recognition method that achieves accuracy good enough for practical applications that concern several ethnic groups.
It is well-known that a human does a better job of recognizing faces within one's own ethnic group. Therefore, it is reasonable to assume that determining the ethnicity of people is an acquired perceptual ability, while determining gender may depend more on instincts. It is commonly observed that people use some set of criteria for determining ethnicity (such as skin color, facial features, etc.). The present invention bases its fundamental approach on an assumption that human perception of ethnicity involves certain high-level reasoning processes. Another assumption is that people use different reasoning processes for different ethnic groups, as some ethnic groups can be largely determined by skin color while the same criterion doesn't apply to other ethnic groups. These assumptions are also based on the observation that the variety of facial appearance due to ethnic differences has certain structure, and there are certain rules that constrain the differences between or among the appearances of facial features.
The present invention aims to solve the problem of ethnicity recognition by images, using the process of probabilistic reasoning based on image features. The reasoning process can be effectively represented using a probabilistic graphical model, where each mode of visual assessment regarding an image feature constitutes a node of the graph, and the process of reasoning are represented by the edges of the graph. Mathematically, the nodes in the probabilistic graphical model represent random variables. In the present case, they quantize certain image features that are relevant to determining ethnicity. The edge between two nodes in the model characterizes the statistical dependence between the image features that the nodes represent. The structure of the graph—what each node represents and the statistical dependence between the nodes—needs to be determined so that the graph can capture the way that certain combinations of image features can probabilistically define an ethnic class. One probabilistic graphical model can be constructed to represent each ethnic group.
In one exemplary embodiment, each graph node captures certain common image features pertaining to certain positions. In this model, the image features are simply local pixel values, or responses of certain filters to specific locations. Each node is assumed to be dependent only on neighboring nodes, so that a simpler probabilistic graphical model, such as a Hidden Markov Model, can be employed.
In another exemplary embodiment, each node represents an unknown biological characteristic that defines ethnicity. The probabilistic graphical model captures the causal process by which the observed appearance features are derived from these unknown biological factors, such as genes.
In another exemplary embodiment, each node represents a correlation between a pair of facial features—such as the relationship between the shape of the eyes and the shape of the nose.
There have been prior attempts for recognizing the demographic category of a person by processing facial images.
The approach by U.S. Pat. No. 6,990,217 of Moghaddam, et al. (hereinafter Moghaddam) addresses the problem of gender recognition by training an SVM using a large number of image instances to make use of image features that distinguish male from female. Moghaddam uses the holistic image features as a single pattern for classification. While the present invention can follow the same approach for ethnicity recognition, it instead takes probabilistic graphical model approaches to explicitly utilize the statistical correlation structure among image features that is meaningful to the given classification problem. In U.S. Pat. No. 7,505,621 of Agrawal, et al. (hereinafter Agrawal), the gender recognition (or demographics recognition in general) is based on comparing individual features, consisting of the indexed and localized feature images and their relative positions. The present invention also makes use of facial features in one of its exemplary embodiments. However, instead of employing a general machine learning approach as in Agrawal, the embodiment of the present invention utilizes the tight correlation structure between facial features under a probabilistic graphical model framework. In U.S. Pat. No. 7,848,548 of Moon, et al. (hereinafter Moon-1), a series of geometric estimations for face localization, three-dimensional facial pose estimations, and face tracking and appearance model building are performed to conduct pose-independent demographics classification. The present invention also makes use of both automatic face localization and automatic facial feature localization so that the whole process of face detection and localization can be performed without any human intervention. While the present invention makes use of such pose estimation for the purpose of normalizing the facial geometry and extracting features, it is not the main focus of the invention. U.S. Pat. No. 8,027,521 of Moon, et al. (hereinafter Moon-2) explicitly extracts facial features and hair features for gender recognition, and the present invention employs a similar approach for facial feature extraction in one of its exemplary embodiment. Unlike Moon-2, however, the present invention aims to represent an ethnic class using the statistical correlations between the extracted features. In “A Method of Gender Classification by Integrating Facial, Hairstyle, and Clothing Images,” in the Proceedings of the 17th International Conference on Pattern Recognition, 2004, by Ueki, et al. (hereinafter Ueki), the gender-specific dress code and hairstyle are exploited for gender recognition, in addition to facial image features. The use of hair features is shared by the present invention. However, in Ueki, the hairstyle extraction is simplified and based on gray-level images due to the dark complexion of the specific ethnicity group, while in the present invention the hair region segmentation can deal with any kinds of skin tone and hair color.
There have been prior attempts for recognizing objects or human faces based on images using probabilistic graphical models.
In U.S. Pat. No. 5,438,630 of Chen, et al. (hereinafter Chen), an HMM (Hidden Markov Model) is employed to encode the text image structures within a given keyword. The present invention takes a similar approach, and employs feature extraction methods that are specific to ethnicity classification. In U.S. Pat. No. 5,787,198 of Agazzi, et al. (hereinafter Agazzi), a pseudo two-dimensional HMM is used to encode the image structure of a given character or a word. U.S. Pat. No. 6,118,890 of Senior (hereinafter Senior) also adopts two-dimensional HMM to encode the image structure of fingerprint patterns. The present invention also adopts a similar architecture in one of its exemplary embodiments, to represent the image structure of a human face more effectively. U.S. Pat. No. 7,171,043 of Nefian (hereinafter Nefian) utilizes a hierarchical HMM for the problem of face recognition, and the present invention utilizes a similar approach to the problem of ethnicity recognition based on facial images. Nefian utilizes observation sequence based on image blocks, while the present invention derives features that are specific to the problem of ethnicity recognition.
There have been prior attempts for extracting image features that suit the purpose of recognizing or classifying objects in images.
In U.S. Pat. Appl. Pub. No. 2004/0066966 of Schneiderman (hereinafter Schneiderman-1) and U.S. Pat. Appl. Pub. No. 2006/0088207 of Schneiderman (hereinafter Schneiderman-2), sub-classifiers are derived for certain object classes as feature extractors or as parts of a cascaded classifier to recognize the object class. In Schneiderman-2, a Bayesian network is employed to aggregate the outputs from sub-classifiers to make a decision in a probabilistic framework. The present invention employs similar approaches for deriving ethnicity-sensitive features. However, the present invention derives the features from facial feature-matched filters in one of its exemplary embodiments, to exploit the domain information. In U.S. Pat. Appl. Pub. No. 2005/0036676 of Heisele (hereinafter Heisele), component classifiers are trained and used to detect each facial feature, and the component classifier outputs along with the spatial constraints are used to recognize faces. The present invention also utilizes filters that are matched to facial components (facial features), but the filters are generated to reflect the variations in size and shape (and sometimes color) and are grouped together to represent feature combinations of an ethnic class.
In summary, the present invention achieves automatic ethnicity recognition based on facial images. Unlike some of the approaches employing holistic image features, the present invention derives ethnicity-sensitive image features based on groups of image features so that each grouping of the image features contributes to more accurate recognition of the ethnic class. It is especially a unique feature of the invention to derive the ethnicity-sensitive image features from image filters that are matched to different colors, sizes, and shapes of facial features—such as eyes, mouth, or complexion. Some of the prior inventions also apply probabilistic graphical models—such as HMM—to face recognition problems, but the present invention applies similar method to the ethnicity recognition problem, utilizing ethnic class-dependent and ethnicity-sensitive image features.